Image display method, image distribution method, image display apparatus, and image distribution apparatus

ABSTRACT

An image display method includes receiving an integrated image in which low-resolution images of a scene from different viewpoints are arranged, displaying the integrated image such that a low-resolution image among the low-resolution images is displayed in a first size, receiving a high-resolution image having a same viewpoint as the low-resolution image, the high-resolution image having a higher resolution than the low-resolution image, displaying the high-resolution image in a second size larger than the first size, and displaying the low-resolution image in a third size larger than the first size after the integrated image is displayed and before the high-resolution image is displayed in the second size.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. continuation application of PCT InternationalPatent Application Number PCT/JP2018/033575 filed on Sep. 11, 2018,claiming the benefit of priority of U.S. Provisional Patent ApplicationNo. 62/557,368 filed on Sep. 12, 2017, the entire contents of which arehereby incorporated by reference.

BACKGROUND 1. Technical Field

The present disclosure relates to an image display method, an imagedistribution method, an image display apparatus, and an imagedistribution apparatus.

2. Description of the Related Art

As a multi-viewpoint video distribution method, Japanese PatentLaid-Open No. 2002-165200 describes a technique by which videos capturedfrom multiple viewpoints are distributed in synchronization withviewpoint movements.

SUMMARY

According to one aspect of the present disclosure, an image displaymethod includes receiving an integrated image in which low-resolutionimages of a scene from different viewpoints are arranged, displaying theintegrated image such that a low-resolution image among thelow-resolution images is displayed in a first size, receiving ahigh-resolution image having a same viewpoint as the low-resolutionimage, the high-resolution image having a higher resolution than thelow-resolution image, displaying the high-resolution image in a secondsize larger than the first size, and displaying the low-resolution imagein a third size larger than the first size after the integrated image isdisplayed and before the high-resolution image is displayed in thesecond size.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description taken in conjunction withthe accompanying drawings that illustrate a specific embodiment of thepresent disclosure.

FIG. 1 is a diagram illustrating an outline of an image distributionsystem according to Embodiment 1;

FIG. 2A is a diagram illustrating an example of an integrated imageaccording to Embodiment 1;

FIG. 2B is a diagram illustrating an example of an integrated imageaccording to Embodiment 1;

FIG. 2C is a diagram illustrating an example of an integrated imageaccording to Embodiment 1;

FIG. 2D is a diagram illustrating an example of an integrated imageaccording to Embodiment 1;

FIG. 3 is a diagram illustrating an example of an integrated imageaccording to Embodiment 1;

FIG. 4 is a diagram illustrating an example of integrated imagesaccording to Embodiment 1;

FIG. 5 is a diagram illustrating a configuration of the imagedistribution system according to Embodiment 1;

FIG. 6 is a block diagram of an integrated video transmission deviceaccording to Embodiment 1;

FIG. 7 is a flowchart of an integrated video generating processaccording to Embodiment 1;

FIG. 8 is a flowchart of a transmission process according to Embodiment1;

FIG. 9 is a block diagram of an image display apparatus according toEmbodiment 1;

FIG. 10 is a flowchart of a receiving process according to Embodiment 1;

FIG. 11 is a flowchart of an image selection process according toEmbodiment 1;

FIG. 12 is a flowchart of an image display process according toEmbodiment 1;

FIG. 13A is a diagram illustrating an example of displaying according toEmbodiment 1;

FIG. 13B is a diagram illustrating an example of displaying according toEmbodiment 1;

FIG. 13C is a diagram illustrating an example of displaying according toEmbodiment 1;

FIG. 14 is a flowchart of a UI process according to Embodiment 1;

FIG. 15 is a diagram illustrating exemplary display of UIs according toEmbodiment 2;

FIG. 16 is a diagram illustrating an example of a viewpoint control UIaccording to Embodiment 2;

FIG. 17 is a diagram illustrating an example of display operation of aviewpoint control UI according to Embodiment 2;

FIG. 18 is a diagram illustrating an example of a viewpoint control UIaccording to Embodiment 2;

FIG. 19 is a diagram illustrating an example of a viewpoint control UIaccording to Embodiment 2;

FIG. 20 is a diagram illustrating exemplary display of a speed controlUI according to Embodiment 2;

FIG. 21 is a diagram illustrating an example of a speed control UIaccording to Embodiment 2;

FIG. 22 is a diagram illustrating an example of a speed control UIaccording to Embodiment 2;

FIG. 23 is a diagram illustrating an example of a speed control UIaccording to Embodiment 2;

FIG. 24 is a diagram illustrating an exemplary display of an indicatoraccording to Embodiment 2;

FIG. 25 is a flowchart of a receiving process according to Embodiment 2;

FIG. 26 is a flowchart of an image display process according toEmbodiment 2;

FIG. 27 is a diagram schematically illustrating an example of switchingthe displayed video according to Embodiment 3;

FIG. 28 is a diagram schematically illustrating an example of switchingthe displayed video according to Embodiment 3;

FIG. 29 is a flowchart of the operations in an image display apparatusand an image distribution apparatus according to Embodiment 3;

FIG. 30 is a diagram illustrating an example of a displayed image andassociated viewpoints according to Embodiment 3;

FIG. 31 is a diagram illustrating an example of a displayed imageaccording to Embodiment 3;

FIG. 32 is a flowchart of an image display process according toEmbodiment 3; and

FIG. 33 is a flowchart of an image display process according toEmbodiment 3.

DETAILED DESCRIPTION OF THE EMBODIMENTS

An image distribution method according to an aspect of the presentdisclosure is an image distribution method in an image distributionsystem in which a plurality of images of a scene seen from differentviewpoints are distributed to a plurality of users, each of whom iscapable of viewing any of the plurality of images. The imagedistribution method includes: generating an integrated image in whichthe plurality of images are arranged in a frame; and distributing theintegrated image to a plurality of image display apparatuses used by theplurality of users.

In this manner, images from multiple viewpoints can be transmitted as asingle integrated image, so that the same integrated image can betransmitted to the multiple image display apparatuses. This can simplifythe system configuration. Using the single-image format can reducechanges to be made on an existing configuration and can also reduce thedata amount of the distributed video with techniques such as an existingimage compression technique.

For example, at least one of the plurality of images included in theintegrated image may be a virtual image generated from a real image.

For example, the plurality of images included in the integrated imagemay have a same resolution.

This facilitates the management of the images. In addition, because themultiple images can be processed in the same manner, the amount ofprocessing can be reduced.

For example, the plurality of images included in the integrated imagemay include images of different resolutions.

In this manner, the quality of images, for example higher-priorityimages, can be improved.

For example, the plurality of images included in the integrated imagemay be images at a same time point.

For example, in the generating, a plurality of integrated imagesincluding the integrated image are generated, and the plurality ofimages included in two or more of the integrated images may be images ata same time point.

In this manner, the number of viewpoints of the images to be distributedcan be increased.

For example, the plurality of images included in the integrated imagemay include images from a same viewpoint at different time points.

This allows the image display apparatuses to display the imagescorrectly even if some of the images are missing due to a communicationerror.

For example, in the distributing, arrangement information indicating anarrangement of the plurality of images in the integrated image may bedistributed to the plurality of image display apparatuses.

For example, in the distributing, information indicating a viewpoint ofeach of the plurality of images in the integrated image may bedistributed to the plurality of image display apparatuses.

For example, in the distributing, time information about each of theplurality of images in the integrated image may be distributed to theplurality of image display apparatuses.

For example, in the distributing, information indicating a switchingorder of the plurality of images in the integrated image may bedistributed to the plurality of image display apparatuses.

An image display method according to an aspect of the present disclosureis an image display method in an image distribution system in which aplurality of images of a scene seen from different viewpoints aredistributed to a plurality of users, each of whom is capable of viewingany of the plurality of images. The image display method includes:receiving an integrated image in which the plurality of images arearranged in a frame; and displaying one of the plurality of imagesincluded in the integrated image.

In this manner, an image of any viewpoint can be displayed by using theimages from multiple viewpoints transmitted as a single integratedimage. This can simplify the system configuration. Using thesingle-image format can reduce changes to be made on an existingconfiguration and can also reduce the data amount of the distributedvideo with techniques such as an existing image compression technique.

An image distribution apparatus according to an aspect of the presentdisclosure is an image distribution apparatus included in an imagedistribution system in which a plurality of images of a scene seen fromdifferent viewpoints are distributed to a plurality of users, each ofwhom is capable of viewing any of the plurality of images. The imagedistribution apparatus includes: a generator that generates anintegrated image in which the plurality of images are arranged in aframe; and a distributor distributes the integrated image to a pluralityof image display apparatuses used by the plurality of users.

In this manner, images from multiple viewpoints can be transmitted as asingle integrated image, so that the same integrated image can betransmitted to the multiple image display apparatuses. This can simplifythe system configuration. Using the single-image format can reducechanges to be made on an existing configuration and can also reduce thedata amount of the distributed video with techniques such as an existingimage compression technique.

An image display apparatus according to an aspect of the presentdisclosure is an image display apparatus included in an imagedistribution system in which a plurality of images of a scene seen fromdifferent viewpoints are distributed to a plurality of users, each ofwhom is capable of viewing any of the plurality of images. The imagedisplay method includes: a receiver that receives an integrated image inwhich the plurality of images are arranged in a frame; and a displaythat displays one of the plurality of images included in the integratedimage.

In this manner, an image of any viewpoint can be displayed by using theimages from multiple viewpoints transmitted as a single integratedimage. This can simplify the system configuration. Using thesingle-image format can reduce changes to be made on an existingconfiguration and can also reduce the data amount of the distributedvideo with techniques such as an existing image compression technique.

An image display method according to an aspect of the present disclosureincludes: displaying at least one of videos of a scene viewed fromdifferent viewpoints, as a displayed video, the videos including atleast one virtual video generated from a real video; displaying a firstuser interface for selecting a viewpoint of the displayed video, andchanging the viewpoint of the displayed video according to an inputprovided via the first user interface; and displaying a second userinterface for changing a playing speed of the displayed video, andchanging the playing speed of the displayed video according to an inputprovided via the second user interface.

Accordingly, the user can easily change the viewpoint being viewed andthe playing speed by using the first user interface and the second userinterface. Therefore, the image display method can improve usability forthe user viewing the video.

For example, the second user interface may be for changing the playingspeed and a playing direction of the displayed video, and in thechanging of the playing speed, the playing speed and the playingdirection of the displayed video may be changed according to an inputprovided via the second user interface.

Accordingly, the user can easily change the playing speed and playingdirection by using the second user interface. Therefore, the imagedisplay method can improve usability for the user viewing the video.

For example, in the second user interface, the playing speed may bechanged by moving an object in a first direction, and the playingdirection may be changed by moving the object in a second directionorthogonal to the first direction.

Accordingly, the user can operate the playing speed and playingdirection intuitively.

For example, the second user interface may comprise a dial, and theplaying speed may be changed according to an amount of rotation of thedial and the playing direction may be changed according to a rotationdirection of the dial.

Accordingly, the user can operate the playing speed and the playingdirection intuitively.

For example, the second user interface may comprise a slide bar, and theplaying speed may be changed according to an amount of sliding on theslide bar and the playing direction may be changed according to asliding direction on the slide bar.

Accordingly, the user can operate the playing speed and the playingdirection intuitively.

For example, when an operation on the second user interface is finished,the playing speed and the playing direction at the time of finishing theoperation may be maintained.

Accordingly, the usability for the user can be improved.

For example, when an operation on the second user interface is finished,the displayed video may be displayed at a predetermined playing speedand in a predetermined playing direction.

Accordingly, the usability for the user can be improved.

For example, when an operation on the second user interface is finished,one of the playing speed and the playing direction may remain in a stateat the time of finishing the operation, and the other of the playingspeed and the playing direction may return to a predetermined state.

Accordingly, the usability for the user can be improved.

For example, at least one of the first user interface or the second userinterface may be transparently displayed when not operated.

Accordingly, it is possible to prevent the first user interface or thesecond user interface from interfering with video viewing, when notbeing operated.

For example, in response to a selection operation at a predeterminedposition on a screen, at least one of the first user interface or thesecond user interface may be displayed at the predetermined position.

Accordingly, the usability for the user can be improved.

For example, in response to a selection operation at any position on ascreen, at least one of the first user interface or the second userinterface may be displayed at the selected position.

Accordingly, the usability for the user can be improved.

For example, in the displaying of the at least one of the videos, aplurality of displayed videos may be displayed, at least one of thefirst user interface or the second user interface may be displayed foreach of the displayed videos, and in response to an operation on the atleast one of the first user interface or the second user interfacecorresponding to one of the videos, at least one of the viewpoint or theplaying speed of the one of the videos may be changed.

Accordingly, the user can control the viewpoint or the playing speed foreach displayed video, and thus usability for the user can be improved.

For example, in the displaying of the at least one of the videos, aplurality of displayed videos may be displayed, and in response to anoperation on at least one of the first user interface or the second userinterface, at least one of the viewpoint or the playing speed of thedisplayed videos may be changed.

Accordingly, the user can collectively operate the displayed videos, andthus usability for the user is improved.

For example, the first user interface may comprise: a path of selectableviewpoint positions; and an object disposed on the path to indicate aviewpoint position, and moving the object to any position on the pathmay cause the viewpoint of the displayed video to be changed to aviewpoint corresponding to the position on the path after moving.

Accordingly, the user can operate the viewpoint of the displayed videointuitively.

For example, the first user interface may comprise: a first operationsection for selecting any of gaze points; and a second operation sectionfor selecting any of viewpoints that focus on a gaze point selected fromthe gaze points.

Accordingly, the user can operate the viewpoint of the displayed videointuitively.

For example, in the first user interface, objects corresponding toviewpoint positions may be disposed on environmental information thattwo-dimensionally or three-dimensionally represents a surroundingenvironment, and selecting any of the objects may cause the displayedvideo to be changed to a video viewed from a viewpoint positioncorresponding to the selected object.

Accordingly, the user can operate the viewpoint of the displayed videointuitively.

An image display apparatus according to an aspect of the presentdisclosure includes: a display configured to display at least one ofvideos of a scene viewed from different viewpoints, as a displayedvideo, the videos including at least one virtual video generated from areal video; a viewpoint changer configured to display a first userinterface for selecting a viewpoint of the displayed video, and changethe viewpoint of the displayed video according to an input provided viathe first user interface; and a playing speed changer configured todisplay a second user interface for changing a playing speed of thedisplayed video, and change the playing speed of the displayed videoaccording to an input provided via the second user interface.

Accordingly, the user can easily change the viewpoint being viewed andthe playing speed by using the first user interface and the second userinterface. Therefore, the image display apparatus can improve usabilityfor the user viewing the video.

An image display method according to an aspect of the present disclosureincludes: receiving an integrated image in which low-resolution imagesof a scene from different viewpoints are arranged in a frame; displayingat least one of the low-resolution images included in the integratedimage received; receiving a high-resolution image which is an image fromthe same viewpoint as and having a higher resolution than a selectedlow-resolution image selected among the low-resolution images; anddisplaying the high-resolution image received.

Accordingly, an increase in the amount of data to be distributed can beprevented using the integrated image in which low-resolution images arearranged in a frame. Since it is possible to display images by switchingbetween the low-resolution image and the high-resolution image, a videoof high image quality can be provided to the user. The image displaymethod can thus improve usability for the user viewing the video whilepreventing an increase in the amount of data to be distributed.

For example, in the displaying of the at least one of the low-resolutionimages, the integrated image may be displayed, and in the displaying ofthe high-resolution image, the high-resolution image may be displayed ina second size larger than a first size of the selected low-resolutionimage as displayed in the integrated image.

For example, the image display method may include: displaying theselected low-resolution image in a third size larger than the first sizeafter the integrated image is displayed; and displaying thehigh-resolution image in the second size after the selectedlow-resolution image is displayed in the third size.

For example, the image display method may include: transitioning fromnormal mode in which the low-resolution images are received anddisplayed to high-resolution mode in which the high-resolution image isreceived and displayed, when a predetermined operation by a user isreceived.

For example, the selected low-resolution image may be a video, and thepredetermined operation may be an operation for pausing the selectedlow-resolution image.

Accordingly, it is possible to provide an image of high image quality tothe user while preventing an increase in the amount of data to bedistributed.

For example, the selected low-resolution image may be a video, thepredetermined operation may be a frame-by-frame advance operation or aslow-motion replay operation for the selected low-resolution image, andin the high-resolution mode, the high-resolution image having a framerate higher than a frame rate of the selected low-resolution image maybe received, and displayed frame by frame or played at a slow speed.

Accordingly, it is possible to provide an image or video of high imagequality to the user in the high-resolution mode.

For example, more viewpoints may be selectable in the high-resolutionmode than in the normal mode.

Accordingly, it is possible to have more viewpoints selectable by theuser in high-resolution mode.

For example, the high-resolution image may be a high-resolution image ofa partial area in the selected low-resolution image.

Accordingly, by displaying only a partial area in high resolution, it ispossible to prevent an increase in the amount of data.

For example, the partial area may be an area containing a face orcharacters.

An image distribution method according to an aspect of the presentdisclosure includes: generating an integrated image in whichlow-resolution images of a scene from different viewpoints are arrangedin a frame; distributing, to an image display apparatus, the integratedimage generated; generating, in response to a request from the imagedisplay apparatus, a high-resolution image which is an image from thesame viewpoint as and having a higher resolution than a selectedlow-resolution image selected among the low-resolution images includedin the integrated image; and distributing, to the image displayapparatus, the high-resolution image generated.

Accordingly, an increase in the amount of data to be distributed can beprevented Using the integrated image in which low-resolution images arearranged in a frame. Since it is possible to display images by switchingbetween the low-resolution image and the high-resolution image, a videoof high image quality can be provided to the user. The imagedistribution method can thus improve usability for the user viewing thevideo while preventing an increase in the amount of data to bedistributed.

For example, the image distribution method may include: transitioningfrom normal mode in which the low-resolution images are distributed tohigh-resolution mode in which the high-resolution image is distributed,when the image display apparatus receives a predetermined operation by auser.

For example, the selected low-resolution image may be a video, and thepredetermined operation may be an operation for pausing the selectedlow-resolution image.

Accordingly, it is possible to provide an image of high image quality tothe user while preventing an increase in the amount of data to bedistributed.

For example, the selected low-resolution image may be a video, thepredetermined operation may be a frame-by-frame advance operation or aslow-motion replay operation for the selected low-resolution image, andin the high-resolution mode, the high-resolution image having a framerate higher than a frame rate of the selected low-resolution image maybe distributed to the image display apparatus.

Accordingly, it is possible to provide an image or video of high imagequality to the user in the high-resolution mode.

For example, more viewpoints may be selectable by the user in thehigh-resolution mode than in the normal mode.

Accordingly, it is possible to have more viewpoints selectable by theuser in high-resolution mode.

For example, the high-resolution image may be a high-resolution image ofa partial area in the selected low-resolution image.

Accordingly, by displaying only a partial area in high resolution, it ispossible to prevent an increase in the amount of data.

For example, the partial area may be an area containing a face orcharacters.

For example, the image distribution method may include: whendistributing the high-resolution image corresponding to the selectedlow-resolution image to the image display apparatus, generating an imagewhich is from a viewpoint close to the viewpoint of the selectedlow-resolution image among the viewpoints of the low-resolution imagesand has a higher resolution than the low-resolution images.

Accordingly, it is possible to reduce the time before video is displayedafter the viewpoint is switched.

An image display apparatus according to an aspect of the presentdisclosure includes: circuitry; and memory, wherein the circuitry, usingthe memory, performs the above-described image display method.

Accordingly, an increase in the amount of data to be distributed can beprevented using the integrated image in which low-resolution images arearranged in a frame. Since it is possible to display images by switchingbetween the low-resolution image and the high-resolution image, a videoof high image quality can be provided to the user. The image displayapparatus can thus improve usability for the user viewing the videowhile preventing an increase in the amount of data to be distributed.

An image distribution apparatus according to an aspect of the presentdisclosure includes: circuitry; and memory, wherein the circuitry, usingthe memory, performs the above-described image distribution method.

Accordingly, an increase in the amount of data to be distributed can beprevented using the integrated image in which low-resolution images arearranged in a frame. Since the image display apparatus can displayimages by switching between the low-resolution image and thehigh-resolution image, a video of high image quality can be provided tothe user. The image distribution method can thus improve usability forthe user viewing the video while preventing an increase in the amount ofdata to be distributed.

Note that these generic or specific aspects may be implemented as asystem, a method, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or may beimplemented as any combination of a system, a method, an integratedcircuit, a computer program, and a recording medium.

Hereinafter, exemplary embodiments will be described in detail withreference to the drawings. Note that each of the following exemplaryembodiments shows a specific example the present disclosure. Thenumerical values, shapes, materials, structural components, thearrangement and connection of the structural components, steps, theprocessing order of the steps, etc. shown in the following embodimentsare mere examples, and thus are not intended to limit the presentdisclosure. Of the structural components described in the followingembodiments, structural components not recited in any one of theindependent claims that indicate the broadest concepts will be describedas optional structural components.

Embodiment 1

This embodiment describes an image distribution system in which videos,including multi-viewpoint videos captured by multi-viewpoint camerasand/or free-viewpoint videos generated using the multi-viewpoint videos,are simultaneously provided to multiple users, who can each change thevideo to view.

With multiple videos such as camera-captured videos and/orfree-viewpoint videos, videos seen from various directions can beacquired or generated. This enables providing videos that meet variousneeds of viewers. For example, an athlete's close-up or long shot can beprovided according to various needs of viewers.

FIG. 1 is a diagram illustrating the overview of an image distributionsystem. For example, a space can be captured using calibrated cameras(e.g., fixed cameras) from multiple viewpoints to three-dimensionallyreconstruct the captured space (three-dimensional space reconstruction).This three-dimensionally reconstructed data can be used to performtracking, scene analysis, and video rendering, thereby generatingfree-viewpoint videos seen from arbitrary viewpoints (free-viewpointcameras). This can realize next-generation wide-area monitoring systemsand free-viewpoint video generation systems.

However, while the system as above can provide various videos, meetingeach viewer's needs requires providing a different video to each viewer.For example, if users watching a sports game in a stadium view videos,there may be thousands of viewers. It is then difficult to have asufficient communication band for distributing a different video to eachof the many viewers. In addition, the distributed video needs to bechanged each time the viewer switches the viewpoint during viewing, andit is difficult to perform this process for each viewer. It is thereforedifficult to realize a system that allows viewers to switch theviewpoint at any point of time.

In light of the above, in the image distribution system according tothis embodiment, two or more viewpoint videos (including camera-capturedvideos and/or free-viewpoint videos) are arranged in a single video (anintegrated video), and the single video and arrangement information aretransmitted to viewers (users). Image display apparatuses (receivingapparatuses) each have the function of displaying one or more viewpointvideos from the single video, and the function of switching thedisplayed video on the basis of the viewer's operation. A system canthus be realized in which many viewers can view videos from differentviewpoints and can switch the viewed video at any point of time.

First, exemplary configurations of the integrated video according tothis embodiment will be described. FIGS. 2A, 2B, 2C, and 2D are diagramsillustrating exemplary integrated images according to this embodiment.An integrated image is an image (a frame) included in the integratedvideo.

As shown in FIGS. 2A to 2D, each of integrated images 151A to 151Dincludes multiple images 152. That is, multiple low-resolution (e.g.,320×180 resolution) images 152 are arranged in each of higher resolution(e.g., 3840×2160 resolution) integrated images 151A to 151D.

Images 152 here are, for example, images at the same time point includedin multiple videos from different viewpoints. For example, in theexample shown in FIG. 2A, nine images 152 are images at the same timepoint included in videos from nine different viewpoints. Note thatimages 152 may include images at different time points.

Images 152 may be of the same resolution as shown in FIGS. 2A and 2B, ormay include images of different resolutions in different patterns asshown in FIGS. 2C and 2D.

For example, the arrangement pattern and the resolutions may bedetermined according to the ratings or the distributor's intension. Asan example, image 152 included in a higher-priority video is set to havea larger size (higher resolution). A higher-priority video here refersto, for example, a video with higher ratings or a video with a higherevaluation value (e.g., a video of a person's close-ups). In thismanner, the image quality of videos in great demand or intended to drawthe viewers' attention can be improved.

Images 152 included in such higher-priority videos may be placed inupper-left areas. The encoding process for streaming distribution or forbroadcasting involves processing for controlling the amount of code.This processing allows the image quality to be more stable in areascloser to the upper-left area, which are the areas scanned earliest. Thequality of the higher-priority images placed in the upper-left areas canthus be stabilized.

Images 152 may be images of the same gaze point seen from differentviewpoints. For example, for a video of a match in a boxing ring, thegaze point may be the center of the ring, and the viewpoints for images152 may be arranged on circumferences about the gaze point.

Images 152 may include images of different gaze points seen from one ormore viewpoints. That is, images 152 may include one or more images of afirst gaze point seen from one or more viewpoints, and one or moreimages of a second gaze point seen from one or more viewpoints. In anexample of a soccer game, the gaze points may be players, and images 152may include images of each player seen from the front, back, right, andleft. For a concert of an idol group, images 152 may include multi-angleimages of the idols, such as each idol's full-length shot and bust shot.

Images 152 may include a 360-degree image for use in technologies suchas VR (Virtual Reality). Images 152 may include an image that reproducesan athlete's sight. Such images may be generated using images 152.

Images 152 may be images included in camera-captured videos actuallycaptured by a camera, or may include one or more free-viewpoint imagesfrom viewpoints inaccessible to a camera, generated through imageprocessing. All images 152 may be free-viewpoint images.

The integrated video may be generated to include integrated images atall time points. Alternatively, integrated images only for some of thetime points in the videos may be generated.

The processing herein may also be performed for still images rather thanvideos.

Now, the arrangement information, which is transmitted along with theintegrated image, will be described. The arrangement information isinformation that defines information about each viewpoint image (image152) in the integrated image and viewpoint switching rules.

The information about each viewpoint image includes viewpointinformation indicating the viewpoint position, or time information aboutthe image. The viewpoint information is information indicating thethree-dimensional coordinates of the viewpoint, or informationindicating a predetermined ID (identification) of the viewpoint positionon a map.

The time information about the viewpoint image may be informationindicating the absolute time, such as the ordinal position of the framein the series of frames, or may be information indicating a relativerelationship with another integrated-image frame.

The information about the viewpoint switching rules includes informationindicating the viewpoint switching order, or grouping information. Theinformation indicating the viewpoint switching order is, for example,table information that defines the relationships among the viewpoints.For example, each image display apparatus 103 can use this tableinformation to determine the viewpoints adjacent to a certain viewpoint.This allows image display apparatus 103 to determine which viewpointimage to use for moving from one viewpoint to an adjacent viewpoint.Image display apparatus 103 can also use this information to readilyrecognize the viewpoint switching order in sequentially changing theviewpoint. This allows image display apparatus 103 to provide animationwith the smoothly switched viewpoint.

A flag may be provided for each viewpoint, indicating that the viewpoint(or the video from the viewpoint) can be used in inter-viewpointtransition for sequential viewpoint movements but the video alone cannotbe displayed.

Images 152 included in the integrated image do not all need to be imagesat the same time point. FIG. 3 is a diagram illustrating an exemplaryconfiguration of integrated image 151E that includes images at differenttime points. For example, as shown in FIG. 3, integrated image 151E attime t includes images 152A at time t, images 152B at time t−1, andimages 152C at time t−2. In the example shown in FIG. 3, images ofvideos from 10 viewpoints at each of the three time points are includedin integrated image 151E.

In this manner, frame loss of the viewpoint videos (images 152A to 152C)could be avoided even if any frame of the integrated video is missing.Specifically, even if integrated image 151E at time t is missing, theimage display apparatus can play the video using images at time tincluded in integrated image 151E at another time point.

FIG. 4 is a diagram illustrating an exemplary configuration ofintegrated images 151F in the case where integrated images at multipletime points include images at the same time point. As shown in FIG. 4,images 152 at time t are included across integrated image 151F at time tand integrated image 151F at time t+1. That is, in the example shown inFIG. 4, each integrated image 151F includes images 152 from 30viewpoints at time t. The two integrated images 151F therefore includeimages 152 from 60 viewpoints in total, at time t. In this manner, anincreased number of viewpoint videos can be provided for a certain timepoint.

The manner of temporally dividing or integrating the frames as above maynot be uniform but may be varied in the video. For example, forimportant scenes such as shoot scenes in a soccer game, the manner shownin FIG. 4 may be used to increase the number of viewpoints; for otherscenes, the integrated image at a given time point may include images152 only at that time point.

Now, the configuration of image distribution system 100 according tothis embodiment will be described. FIG. 5 is a block diagram of imagedistribution system 100 according to this embodiment. Image distributionsystem 100 includes cameras 101, image distribution apparatus 102, andimage display apparatuses 103.

Cameras 101 generate a group of camera-captured videos, which aremulti-viewpoint videos. The videos may be synchronously captured by allcameras. Alternatively, time information may be embedded in the videos,or index information indicating the frame order may be attached to thevideos, so that image distribution apparatus 102 can identify images(frames) at the same time point. Note that one or more camera-capturedvideos may be generated by one or more cameras 101.

Image distribution apparatus 102 includes free-viewpoint videogeneration device 104 and integrated video transmission device 105.Free-viewpoint video generation device 104 uses one or morecamera-captured videos from cameras 101 to generate one or morefree-viewpoint videos seen from virtual viewpoints. Free-viewpoint videogeneration device 104 sends the generated one or more free-viewpointvideos (a group of free-viewpoint videos) to integrated videotransmission device 105.

For example, free-viewpoint video generation device 104 may use thecamera-captured videos and positional information about the videos toreconstruct a three-dimensional space, thereby generating athree-dimensional model. Free-viewpoint video generation device 104 maythen use the generated three-dimensional model to generate afree-viewpoint video. Free-viewpoint video generation device 104 mayalso generate a free-viewpoint video by using images captured by two ormore cameras to interpolate camera-captured videos.

Integrated video transmission device 105 uses one or morecamera-captured videos and/or one or more free-viewpoint videos togenerate an integrated video in which each frame includes multipleimages. Integrated video transmission device 105 transmits, to imagedisplay apparatuses 103, the generated integrated video and arrangementinformation indicating information such as the positional relationshipsamong the videos in the integrated video.

Each of image display apparatuses 103 receives the integrated video andthe arrangement information transmitted by image distribution apparatus102 and displays, to a user, at least one of the viewpoint videosincluded in the integrated video. Image display apparatus 103 switchesthe displayed viewpoint video in response to a UI operation. Thisrealizes an interactive video switching based on the user's operations.Image display apparatus 103 feeds back viewing information, indicatingthe currently used viewpoint or currently viewed viewpoint video, toimage distribution apparatus 102. Note that image distribution system100 may include one or more image display apparatuses 103.

Now, the configuration of integrated video transmission device 105 willbe described. FIG. 6 is a block diagram of integrated video transmissiondevice 105. Integrated video transmission device 105 includes integratedvideo generator 201, transmitter 202, and viewing information analyzer203.

Integrated video generator 201 generates an integrated video from two ormore videos (camera-captured videos and/or free-viewpoint videos) andgenerates arrangement information about each video in the integratedvideo.

Transmitter 202 transmits the integrated video and the arrangementinformation generated by integrated video generator 201 to one or moreimage display apparatuses 103. Transmitter 202 may transmit theintegrated video and the arrangement information to image displayapparatuses 103 either as one stream or through separate paths. Forexample, transmitter 202 may transmit, to image display apparatuses 103,the integrated video through a broadcast wave and the arrangementinformation through network communication.

Viewing information analyzer 203 aggregates viewing information (e.g.,information indicating the viewpoint video currently displayed on eachimage display apparatus 103) transmitted from one or more image displayapparatuses 103. Viewing information analyzer 203 passes the resultingstatistical information (e.g., the ratings) to integrated videogenerator 201. Integrated video generator 201 uses this statisticalinformation as referential information in integrated-video generation.

Transmitter 202 may stream the integrated video and the arrangementinformation or may transmit them as a unit of sequential video frames.

As a rendering effect preceding the initial view of the distributedvideo, image distribution apparatus 102 may generate a video in whichthe view is sequentially switched from a long-shot view to the initialview, and may distribute the generated video. This can provide, e.g., asa lead-in to a replay, a scene allowing the viewers to grasp spatialinformation, such as the position or posture with respect to the initialviewpoint. This processing may be performed in image display apparatuses103 instead. Alternatively, image distribution apparatus 102 may sendinformation indicating the switching order and switching timings ofviewpoint videos to image display apparatuses 103, which may then switchthe displayed viewpoint video according to the received information tocreate the above-described video.

Now, the flow of operations in integrated video generator 201 will bedescribed. FIG. 7 is a flowchart of the process of generating theintegrated video by integrated video generator 201.

First, integrated video generator 201 acquires multi-viewpoint videos(S101). The multi-viewpoint videos include two or more videos in total,including camera-captured videos and/or free-viewpoint videos generatedthrough image processing, such as a free-viewpoint video generationprocessing or morphing processing. The camera-captured videos do notneed to be directly transmitted from cameras 101 to integrated videogenerator 201. Rather, the videos may be saved in some other storagebefore being input to integrated video generator 201; in this case, asystem utilizing archived past videos, instead of real-time videos, canbe constructed.

Integrated video generator 201 determines whether there is viewinginformation from image display apparatuses 103 (S102). If there isviewing information (Yes at S102), integrated video generator 201acquires the viewing information (e.g., the ratings of each viewpointvideo) (S103). If viewing information is not to be used, the process atsteps S102 and S103 is skipped.

Integrated video generator 201 generates an integrated video from theinput multi-viewpoint videos (S104). First, integrated video generator201 determines how to divide the frame area for arranging the viewpointvideos in the integrated video. Here, integrated video generator 201 mayarrange all videos in the same resolution as shown in FIGS. 2A and 2B,or the videos may vary in resolution as shown in FIGS. 2C and 2D.

If the videos are set to have the same resolution, the processing loadcan be reduced because the videos from all viewpoints can be processedin the same manner in subsequent stages. By contrast, if the videos varyin resolution, the image quality of higher-priority videos (such as avideo from a viewpoint recommended by the distributor) can be improvedto provide a service tailored to the viewers.

As shown in FIG. 3, an integrated image at a certain time point mayinclude multi-viewpoint images at multiple time points. As shown in FIG.4, integrated images at multiple time points may include multi-viewpointimages at the same time point. The former way can ensure redundancy inthe temporal direction, thereby providing stable video viewingexperiences even under unstable communication conditions. The latter waycan provide an increased number of viewpoints.

Integrated video generator 201 may vary the dividing scheme according tothe viewing information acquired at step S103. Specifically, a viewpointvideo with higher ratings may be placed in a higher resolution area sothat the video is rendered with a definition higher than the definitionof the other videos.

Integrated video generator 201 generates arrangement information. Thearrangement information includes the determined dividing scheme andinformation associating the divided areas with viewpoint informationabout the respective input videos (i.e., information indicating whichviewpoint video is placed in which area). Here, integrated videogenerator 201 may further generate transition information indicatingtransitions between the viewpoints, and grouping information presentinga video group for each player.

On the basis of the generated arrangement information, integrated videogenerator 201 generates the integrated video from the two or more inputvideos.

Finally, integrated video generator 201 encodes the integrated video(S105). This process is not required if the communication band issufficient. Integrated video generator 201 may set each video as anencoding unit. For example, integrated video transmission device 105 mayset each video as a slice or tile in H.265/HEVC. The integrated videomay then be encoded in a manner that allows each video to beindependently decoded. This allows only one viewpoint video to bedecoded in a decoding process, so that the amount of processing in imagedisplay apparatuses 103 can be reduced.

Integrated video generator 201 may vary the amount of code assigned toeach video according to the viewing information. Specifically, for anarea in which a video with high ratings is placed, integrated videogenerator 201 may improve the image quality by reducing the value of aquantization parameter.

Integrated video generator 201 may make the image quality (e.g., theresolution or the quantization parameter) uniform for a certain group(e.g., viewpoints focusing on the same player as the gaze point, orconcyclic viewpoints). In this manner, the degree of change in imagequality at the time of viewpoint switching can be reduced.

Integrated video generator 201 may process the border areas and theother areas differently. For example, a deblocking filter may not beused for the borders between the viewpoint videos.

Now, a process in transmitter 202 will be described. FIG. 8 is aflowchart of a process performed by transmitter 202.

First, transmitter 202 acquires the integrated video generated byintegrated video generator 201 (S201). Transmitter 202 then acquires thearrangement information generated by integrated video generator 201(S202). If there are no changes in the arrangement information,transmitter 202 may reuse the arrangement information used for theprevious frame instead of acquiring new arrangement information.

Finally, transmitter 202 transmits the integrated video and thearrangement information acquired at steps 5201 and 5202 (S203).Transmitter 202 may broadcast these information items, or may transmitthese information items using one-to-one communication. Transmitter 202does not need to transmit the arrangement information for each frame butmay transmit the arrangement information when the video arrangement ischanged. Transmitter 202 may also transmit the arrangement informationat regular intervals (e.g., every second). The former way can minimizethe amount of information to be transmitted. The latter way allows imagedisplay apparatuses 103 to regularly acquire correct arrangementinformation; image display apparatuses 103 can then address a failure ininformation acquisition due to communication conditions or can addressacquisition of an in-progress video.

Transmitter 202 may transmit the integrated video and the arrangementinformation as interleaved or as separate pieces of information.Transmitter 202 may transmit the integrated video and the arrangementinformation through a communication path such as the Internet, orthrough a broadcast wave. Transmitter 202 may also combine thesetransmission schemes. For example, transmitter 202 may transmit theintegrated video through a broadcast wave and transmit the arrangementinformation through a communication path.

Now, the configuration of each image display apparatus 103 will bedescribed. FIG. 9 is a block diagram of image display apparatus 103.Image display apparatus 103 includes receiver 301, viewpoint videoselector 302, video display 303, UI device 304, UI controller 305, andviewing information transmitter 306.

Receiver 301 receives the integrated video and the arrangementinformation transmitted by integrated video transmission device 105.Receiver 301 may have a buffer or memory for saving received items suchas videos.

Viewpoint video selector 302 selects one or more currently displayedviewpoint videos from the received integrated video using thearrangement information and selected-viewpoint information indicatingthe currently displayed viewpoint video(s). Viewpoint video selector 302outputs the selected viewpoint video(s).

Video display 303 displays the one or more viewpoint videos selected byviewpoint video selector 302.

UI device 304 interprets the user's input operation and displaying a UI(User Interface). The input operation may be performed with an inputdevice such as a mouse, keyboard, controller, or touch panel, or with atechnique such as speech recognition or camera-based gesturerecognition. Image display apparatus 103 may be a device (e.g., asmartphone or a tablet terminal) equipped with a sensor such as anaccelerometer, so that the tilt and the like of image display apparatus103 may be detected to acquire an input operation accordingly.

On the basis of an input operation acquired by UI device 304, UIcontroller 305 outputs information for switching the viewpoint video(s)being displayed. UI controller 305 also updates the content of the UIdisplayed on UI device 304.

On the basis of the selected-viewpoint information indicating theviewpoint video(s) selected by viewpoint video selector 302, viewinginformation transmitter 306 transmits viewing information to integratedvideo transmission device 105. The viewing information is informationabout the current viewing situations (e.g., index information about theselected viewpoint).

FIG. 10 is a flowchart indicating operations in receiver 301. First,receiver 301 receives information transmitted by integrated videotransmission device 105 (S301). In streaming play mode, the transmittedinformation may be input to receiver 301 via a buffer capable of savingvideo for a certain amount of time.

If receiver 301 receives the video as a unit of sequential video frames,receiver 301 may store the received information in storage such as anHDD or memory. The video may then be played and paused as requested by acomponent such as viewpoint video selector 302 in subsequent processes.This allows the user to pause the video at a noticeable scene (e.g., animpactful moment in a baseball game) to view the scene from multipledirections. Alternatively, image display apparatus 103 may generate sucha video.

If the video is paused while being streamed, image display apparatus 103may skip the part of the video of the paused period and stream thesubsequent part of the video. Image display apparatus 103 may also skipor fast-forward some of the frames of the buffered video to generate adigest video shorter than the buffered video, and display the generateddigest video. In this manner, the video to be displayed after a lapse ofa certain period can be aligned with the streaming time.

Receiver 301 acquires an integrated video included in the receivedinformation (S302). Receiver 301 determines whether the receivedinformation includes arrangement information (S303). If it is determinedthat the received information includes arrangement information (Yes atS303), receiver 301 acquires the arrangement information in the receivedinformation (S304).

FIG. 11 is a flowchart indicating a process in viewpoint video selector302. First, viewpoint video selector 302 acquires the integrated videooutput by receiver 301 (S401). Viewpoint video selector 302 thenacquires the arrangement information output by receiver 301 (S402).

Viewpoint video selector 302 acquires, from UI controller 305, theselected-viewpoint information for determining the viewpoint for display(S403). Instead of acquiring the selected-viewpoint information from UIcontroller 305, viewpoint video selector 302 itself may manageinformation such as the previous state. For example, viewpoint videoselector 302 may select the viewpoint used in the previous state.

On the basis of the arrangement information acquired at step S402 andthe selected-viewpoint information acquired at step S403, viewpointvideo selector 302 acquires a corresponding viewpoint video from theintegrated video acquired at step S401 (S404). For example, viewpointvideo selector 302 may clip out a viewpoint video from the integratedvideo so that a desired video is displayed on video display 303.Alternatively, video display 303 may display a viewpoint video byenlarging the area of the selected viewpoint video in the integratedvideo to fit the area into the display area.

For example, the arrangement information is a binary image of the sameresolution as the integrated image, where 1 is set in the borderportions and 0 is set in the other portions. The binary image isassigned sequential IDs starting at the upper-left corner. Viewpointvideo selector 302 acquires the desired video by extracting a video inthe area having an ID corresponding to the viewpoint indicated in theselected-viewpoint information. The arrangement information does notneed to be an image but may be text information indicating thetwo-dimensional viewpoint coordinates and the resolutions.

Viewpoint video selector 302 outputs the viewpoint video acquired atstep S404 to video display 303 (S405).

Viewpoint video selector 302 also outputs the selected-viewpointinformation indicating the currently selected viewpoint to viewinginformation transmitter 306 (S406).

Not only one video but videos from multiple viewpoints may be selectedon the basis of the selected-viewpoint information. For example, a videofrom one viewpoint and videos from neighboring viewpoints may beselected, or a video from one viewpoint and videos from other viewpointssharing the gaze point with that video may be selected. For example, ifthe selected-viewpoint information indicates a viewpoint focusing on aplayer A from the front of the player A, viewpoint video selector 302may select a viewpoint video in which the player A is seen from a sideor the back, in addition to the front-view video.

For viewpoint video selector 302 to select multiple viewpoints, theselected-viewpoint information may simply indicate the multipleviewpoints to be selected. The selected-viewpoint information may alsoindicate a representative viewpoint, and viewpoint video selector 302may estimate other viewpoints based on the representative viewpoint. Forexample, if the representative viewpoint focuses on a player B,viewpoint video selector 302 may select videos from viewpoints focusingon other players C and D, in addition to the representative-viewpointvideo.

The initial value of the selected-viewpoint information may be embeddedin the arrangement information or may be predetermined. For example, aposition in the integrated video (e.g., the upper-left corner) may beused as the initial value. The initial value may also be determined byviewpoint video selector 302 according to the viewing situations such asthe ratings. The initial value may also be automatically determinedaccording to the user's preregistered preference in camera-capturedsubjects, which are identified with face recognition.

FIG. 12 is a flowchart illustrating operations in video display 303.First, video display 303 acquires the one or more viewpoint videosoutput by viewpoint video selector 302 (S501). Video display 303displays the viewpoint video(s) acquired at step S501 (S502).

FIGS. 13A, 13B, and 13C are diagrams illustrating exemplary display ofvideos on video display 303. For example, as shown in FIG. 13A, videodisplay 303 may display one viewpoint video 153 alone. Video display 303may also display multiple viewpoint videos 153. For example, in theexample shown in FIG. 13B, video display 303 displays all viewpointvideos 153 in the same resolution. As shown in FIG. 13C, video display303 may also display viewpoint videos 153 in different resolutions.

Image display apparatus 103 may save the previous frames of theviewpoint videos, with which an interpolation video may be generatedthrough image processing when the viewpoint is to be switched, and thegenerated interpolation video may be displayed at the time of viewpointswitching. Specifically, when the viewpoint is to be switched to anadjacent viewpoint, image display apparatus 103 may generate anintermediate video through morphing processing and display the generatedintermediate video. This can produce a smooth viewpoint change.

FIG. 14 is a flowchart illustrating a process in UI device 304 and UIcontroller 305. First, UI controller 305 determines an initial viewpoint(S601) and sends initial information indicating the determined initialviewpoint to UI device 304 (S602).

UI controller 305 then waits for an input from UI device 304 (S603).

If the user's input information is received from UI device 304 (Yes atS603), UI controller 305 updates the selected-viewpoint informationaccording to the input information (S604) and sends the updatedselected-viewpoint information to UI device 304 (S605).

UI device 304, first, receives the initial information from UIcontroller 305 (S701). UI device 304 displays a UI according to theinitial information (S702). As the UI, UI device 304 displays any one ora combination of two or more of the following UIs. For example, UIdevice 304 may display a selector button for switching the viewpoint. UIdevice 304 may also display a projection, like map information,indicating the two-dimensional position of each viewpoint. UI device 304may also display a representative image of the gaze point of eachviewpoint (e.g., a face image of each player).

UI device 304 may change the displayed UI according to the arrangementinformation. For example, if the viewpoints are concyclically arranged,UI device 304 may display a jog dial; if the viewpoints are arranged ona straight line, UI device 304 may display a UI for performing slide orflick operations. This enables the viewer's intuitive operations. Notethat the above examples are for illustration, and a UI for performingslide operations may be used for a concyclic camera arrangement as well.

UI device 304 determines whether the user's input is provided (S703).This input operation may be performed via an input device such as akeyboard or a touch panel, or may result from interpreting an output ofa sensor such as an accelerometer. The input operation may also usespeech recognition or gesture recognition. If the videos arranged in theintegrated video include videos of the same gaze point with differentzoom factors, a pinch-in or pinch-out operation may cause the selectedviewpoint to be transitioned to another viewpoint.

If the user's input is provided (Yes at S703), UI device 304 generatesinput information for changing the viewpoint on the basis of the user'sinput and sends the generated input information to UI controller 305(S704). UI device 304 then receives the updated selected-viewpointinformation from UI controller 305 (S705), updates UI informationaccording to the received selected-viewpoint information (S706), anddisplays a UI based on the updated UI information (S702).

As above, image distribution apparatus 102 is included in imagedistribution system 100 in which images of a scene seen from differentviewpoints are distributed to users, who can each view any of theimages. Image distribution apparatus 102 generates an integrated image(such as integrated image 151A) having images 152 arranged in a frame.Image distribution apparatus 102 distributes the integrated image toimage display apparatuses 103 used by the users.

In this manner, images from multiple viewpoints can be transmitted as asingle integrated image, so that the same integrated image can betransmitted to the multiple image display apparatuses 103. This cansimplify the system configuration. Using the single-image format canreduce changes to be made on an existing configuration and can alsoreduce the data amount of the distributed video with techniques such asan existing image compression technique.

At least one of the images included in the integrated image may be avirtual image (free-viewpoint image) generated from a real image.

As shown in FIGS. 2A and 2B, images 152 included in integrated image151A or 151B may have the same resolution. This facilitates themanagement of images 152. In addition, because multiple images 152 canbe processed in the same manner, the amount of processing can bereduced.

Alternatively, as shown in FIGS. 2C and 2D, images 152 included inintegrated image 151C or 151D may include images 152 of differentresolutions. In this manner, the quality of images 152, for examplehigher-priority images, can be improved.

The images included in the integrated image may be images at the sametime point. As shown in FIG. 4, images 152 included in two or moreintegrated images 151F may be images at the same time point. In thismanner, the number of viewpoints to be distributed can be increased.

As shown in FIG. 3, images 152A, 152B, and 152C included in integratedimage 151E may include images from the same viewpoint at different timepoints. This allows image display apparatuses 103 to display the imagescorrectly even if some of the images are missing due to a communicationerror.

Image distribution apparatus 102 may distribute arrangement informationindicating the arrangement of the images in the integrated image toimage display apparatuses 103. Image distribution apparatus 102 may alsodistribute information indicating the viewpoint of each of the images inthe integrated image to image display apparatuses 103. Imagedistribution apparatus 102 may also distribute time information abouteach of the images in the integrated image to image display apparatuses103. Image distribution apparatus 102 may also distribute informationindicating the switching order of the images in the integrated image toimage display apparatuses 103.

Image display apparatuses 103 are included in image distribution system100. Each image display apparatus 103 receives an integrated image (suchas integrated image 151A) having images 152 arranged in a frame. Imagedisplay apparatus 103 displays one of images 152 included in theintegrated image.

In this manner, an image of any viewpoint can be displayed by using theimages from multiple viewpoints transmitted as a single integratedimage. This can simplify the system configuration. Using thesingle-image format can reduce changes to be made on an existingconfiguration and can also reduce the data amount of the distributedvideo with techniques such as an existing image compression technique.

Image display apparatus 103 may receive arrangement informationindicating the arrangement of the images in the integrated image, anduse the received arrangement information to acquire image 152 from theintegrated image.

Image display apparatus 103 may receive information indicating theviewpoint of each of the images in the integrated image, and use thereceived information to acquire image 152 from the integrated image.

Image display apparatus 103 may receive time information about each ofthe images in the integrated image, and use the received timeinformation to acquire image 152 from the integrated image.

Image display apparatus 103 may receive information indicating theswitching order of the images in the integrated image, and use thereceived information to acquire image 152 from the integrated image.

Embodiment 2

In this embodiment, details of UIs in image display apparatus 103 willbe described. Image display apparatus 103 is configured as shown in FIG.9.

In this embodiment, on the basis of the input information acquired by UIdevice 304, UI controller 305 outputs information for switching theviewpoint video being displayed and for changing the playing speed andthe playing direction.

Viewpoint video selector 302 outputs the viewpoint video according tothe playing direction and the playing speed output from UI controller305.

FIG. 15 is a diagram showing exemplary display of UIs in image displayapparatus 103. As shown in FIG. 15, UI device 304 displays indicator 403that presents information for a user, such as information indicatingwhether the viewpoint is switchable, viewpoint control UI 401 forswitching the displayed video, and speed control UI 402 for controllingthe playing speed, including playing and pausing. At least one ofviewpoint control UI 401, speed control UI 402, and indicator 403 may bedisplayed to overlap the video or may be displayed in an area that doesnot overlap the video.

Video display 303 displays, as a displayed video, at least one ofvideos. Here, the videos are multi-viewpoint videos of a scene viewedfrom different viewpoints. For example, the videos include at least onevirtual video (free-viewpoint video) generated from real video.

Viewpoint control UI 401 (a first user interface) is a UI for selectingthe viewpoint of the displayed video. Viewpoint video selector 302changes the viewpoint of the displayed video according to the user'sinput provided via viewpoint control UI 401.

Speed control UI 402 is a UI for changing the playing speed and theplaying direction of the displayed video. Viewpoint video selector 302changes the playing speed and the playing direction of the displayedvideo according to the user's input provided via speed control UI 402.

Viewpoint control UI 401 will be described below. FIG. 16 is a diagramshowing exemplary display of viewpoint control UI 401. FIG. 17 is adiagram showing an example of switching the display of viewpoint controlUI 401 with a tap operation. UI device 304 may display viewpoint controlUI 401 at all times or only when viewpoint control UI 401 is operated.For example, image display apparatus 103 may display viewpoint controlUI 401 usually in transparent form, but in normal form (innontransparent form) if viewpoint control UI 401 is operated.

Alternatively, as shown in FIG. 17, image display apparatus 103 may hideviewpoint control UI 401 when no operation is performed, but displayviewpoint control UI 401 if a selection operation is performed in apredetermined specific area. Here, the selection operation may be a tapor click operation, for example. The specific area may be an area within100 pixels from the right edge of the screen, for example.

That is, image display apparatus 103 may display viewpoint control UI401 at a predetermined position on the screen in response to a selectionoperation at the predetermined position.

Image display apparatus 103 may display viewpoint control UI 401 at anyselected position on the screen in response to the user's selectionoperation at that position. For example, viewpoint control UI 401 may bedisplayed at a position centered around a point at which the user hasperformed an operation. Alternatively, UI device 304 may displayviewpoint control UI 401 at a predetermined fixed position in responseto the user's selection operation at any position on the screen.

UI device 304 may display viewpoint control UI 401 only when a selectionoperation such as a tap operation is being performed, or may switchbetween displaying and hiding viewpoint control UI 401 upon everyselection operation. UI device 304 may keep viewpoint control UI 401displayed for a predetermined time period after a selection operationand hide viewpoint control UI 401 after a lapse of the predeterminedtime period.

The above control is applicable not only to switching between displayingand hiding but also to switching between the normal display and thetransparent display.

As shown in FIG. 16, viewpoint control UI 401 includes path 411 ofselectable viewpoint positions, and object 412 disposed on path 411 toindicate a viewpoint position. Moving object 412 to any position on path411 causes the viewpoint of the displayed video to be changed to aviewpoint corresponding to the position of moved object 412 on path 411.

Here, object 412 is allowed to be moved only on path 411. Object 412 maybe operated in any manner. For example, object 412 may be moved bytapping object 412 and dragging the tapped position. Alternatively,Object 412 may be moved by tapping any position on path 411 or near path411 to cause object 412 to be moved to the tapped position on path 411or to a position on path 411 near the tapped position.

In FIG. 16 where path 411 is circular, the inside of circular path 411is captured, for example. That is, the gaze points exist inside thecircle. The user can therefore intuitively switch the viewpoint on thepredefined shape with a single operation. Path 411 is not limited tocircular but may be in elliptic, rectangular, or other shapes. Path 411does not need to have a shape surrounding the gaze points but may belinear. Further, rather than the inside of the circle, the outside ofthe circle may be captured.

FIGS. 18 and 19 are diagrams showing variations of viewpoint control UI401. Viewpoint control UI 401A shown in FIG. 18 includes first operationsection 421 for selecting any of gaze points, and second operationsection 422 for selecting any of viewpoints that focus on the gaze pointselected from the gaze points.

In the example shown in FIG. 18, the gaze points are players A to C.First operation section 421 includes buttons corresponding to therespective players. Pressing a button corresponding to a certain playercauses second operation section 422 to display this player 423 andobjects 424 representing viewpoints that focus on this player 423. Ifthe user performs an operation of selecting one of objects 424, thedisplayed video is changed to a video of the viewpoint corresponding toselected object 424. In the example shown in FIG. 18, each object 424 isa camera icon representing a viewpoint position and a capturingdirection (a camera orientation).

The gaze points may not be players but may be moving objects (such ashumans, animals, and vehicles), still objects, or any points or areas ina three-dimensional space.

Viewpoint control UI 401B shown in FIG. 19 includes environmentalinformation 431 that two-dimensionally or three-dimensionally representsthe surrounding environment. Objects 432 corresponding to viewpointpositions are disposed on environmental information 431. Selecting anyof objects 432 causes the displayed video to be changed to a video ofthe viewpoint position corresponding to the selected object 432. In theexample shown in FIG. 19, each object 432 is a camera icon representinga viewpoint position and a capturing direction (a camera orientation).

Although environmental information 431 is a top view of a court in theexample shown in FIG. 19, environmental information 431 may be in otherforms such as map information or a floor plan. Environmental information431 may also be other views such as a perspective view, rather than atop view.

The above-described specific examples of viewpoint control UI 401 areexemplary; any UIs capable of specifying a video to be displayed out ofthe integrated video may be employed.

For example, although the above description illustrates the example inwhich the two-dimensional position of a viewpoint is selected, theheight may be selected in addition to the two-dimensional position. Inthis case, a side view for selecting the height may be displayed inaddition to the top view, or an operation section for switching betweenthe top view and the side view may be provided. Alternatively, aperspective view that enables the user to see the two-dimensionalposition and the height may be used, or an operation section forchanging the angle of environmental information 431 and the like may beprovided.

Multiple videos may be displayed, for example on the screen partitionedinto sub-screens. UI device 304 may then display viewpoint control UI401 for each video, so that the viewpoints of the videos may beindividually changed. That is, viewpoint control UI 401 is displayed foreach of the displayed videos. In response to an operation oncorresponding viewpoint control UI 401, the viewpoint of thecorresponding video is changed.

Alternatively, in synchronization with change of the viewpoint of onevideo, the viewpoints of the other videos may be switched. That is, theviewpoints of the displayed videos may be changed in response to anoperation on one viewpoint control UI 401. If the videos are to besynchronously switched in this manner, only one viewpoint control UI 401may be displayed or multiple viewpoint control UIs 401 may be displayed.

For example, in the UI shown in FIG. 16, moving the viewpoint positionrightward causes the viewpoints of the respective videos to be movedrightward. If videos are being displayed that show a player A viewedfrom multiple viewpoints and an operation is performed for switching thetarget player from the player A to a player B, videos showing the playerB viewed from multiple viewpoints are displayed.

Now, speed control UI 402 will be described. FIG. 20 is a diagramshowing exemplary display of speed control UI 402. FIG. 21 is a diagramfor describing operations with speed control UI 402.

Speed control UI 402 is a UI for pausing the displayed video, changingthe playing speed, and changing the playing direction (forward andbackward).

Speed control UI 402 shown in FIG. 21 is a UI based on stick operationsand includes object 441. The user can move object 441 in a firstdirection (the vertical direction in FIG. 21) to change the playingspeed. The user can move object 441 in a second direction (thehorizontal direction in FIG. 21) orthogonal to the first direction tochange the playing direction.

Thus, video playing is controlled by changing the position of roundobject 441 at the center. Specifically, the playing speed is controlledin the vertical direction (the upward direction corresponds toacceleration and the downward direction corresponds to deceleration).The speed depends on the position in the vertical direction; the playingspeed is higher if object 441 is positioned higher, and the playingspeed is lower if object 441 is positioned lower. The temporal directionis controlled in the horizontal direction (the leftward directioncorresponds to backward playing and the rightward direction correspondsto forward playing). The displayed video is paused if object 441 ispositioned at the center.

FIGS. 22 and 23 are diagrams showing variations of speed control UI 402.Speed control UI 402A shown in FIG. 22 is a dial-shaped UI in whichvideo playing is controlled according to the rotation direction and theamount of rotation. That is, speed control UI 402A includes a dial suchthat the playing speed is changed according to the amount of rotation ofthe dial, and the playing direction is changed according to the rotationdirection of the dial.

Specifically, the video is played backward if the dial is rotatedleftward, and played forward if the dial is rotated rightward. Thelarger the amount of rotation is, the faster the video is played. Thevideo is paused if the amount of rotation from the initial position iszero.

Speed control UI 402B shown in FIG. 23 is a slide bar UI in which videoplaying is controlled according to the amount of sliding and the slidingdirection. That is, speed control UI 402B includes a slide bar such thatthe playing speed is changed according to the amount of sliding on theslide bar, and the playing direction is changed according to the slidingdirection on the slide bar. The video is paused if the slider is at theinitial position.

As with viewpoint control UI 401, speed control UI 402 may be displayedat all times, or may be hidden or transparently displayed when notoperated.

As with viewpoint control UI 401, if multiple videos are displayed,speed control UI 402 may be displayed for each of the displayed videos.In response to an operation on speed control UI 402, the playing speedand the playing direction of the corresponding video may be changed.Alternatively, the playing speed and the playing direction of thedisplayed videos may be changed in response to an operation on one speedcontrol UI 402.

When the user stops the operation on speed control UI 402, the playingof the displayed video may be controlled in any of the followingmanners.

When the operation on speed control UI 402 is finished, the displayedvideo may be displayed at a predetermined playing speed and in apredetermined playing direction. For example, the video is automaticallyset to be played forward at a predetermined standard playing speed.

Alternatively, the settings at the time of stopping the operation may bemaintained. That is, when the operation on speed control UI 402 isfinished, the playing speed and the playing direction at the time offinishing the operation may be maintained.

Alternatively, either one of the playing speed and the temporaldirection may be kept at the setting at the time of stopping theoperation, and the other one may be automatically set to a standardsetting. That is, when the operation on speed control UI 402 isfinished, one of the playing speed and the playing direction may remainin the state at the time of finishing the operation, and the other mayreturn to a predetermined state. For example, the playing direction maybe maintained and the playing speed may be set to a predetermined speed.Different playing speeds may be predetermined for forward playing andbackward playing.

Alternatively, when the operation is stopped, the playing of thedisplayed video may be paused with an image at the time of stopping theoperation being displayed.

When an operation for backward playing is performed, the video may beplayed backward while the display viewpoint at the time of the operationis maintained. Alternatively, image display apparatus 103 may saveinformation about previous viewpoint switching and, on the basis of thisinformation, may play the video backward while changing the viewpoint inthe reverse viewpoint-switching order.

Now, indicator 403 will be described. FIG. 24 is a diagram showingexemplary display of indicator 403. Indicator 403 is used for notifyingthe user (viewer) whether the viewpoint is switchable.

FIG. 25 is a flowchart showing an exemplary process in image displayapparatus 103. Processing at steps S801 to S803 is the same as theprocessing at steps S301 to S303 shown in FIG. 10. For example, as shownin FIG. 25, if the received information includes the arrangementinformation (Yes at S803), image display apparatus 103 notifies the userthat the viewpoint is switchable (S804). For example, as shown in FIG.24, indicator 403 displays an icon indicating that the viewpoint isswitchable.

Rather than displaying the predetermined icon in the specific area,other manners may be used to notify the user whether the viewpoint isswitchable. For example, viewpoint control UI 401 or the like may bedisplayed if the viewpoint is switchable, while not be displayed if theviewpoint is unswitchable. Sounds such as sound effects may be providedfor notifying that the viewpoint is switchable or whether the viewpointis switchable. Image display apparatus 103 may vibrate to notify thatthe viewpoint is switchable or whether the viewpoint is switchable.Thus, any presentation manner may be used that can notify the user thatthe viewpoint is switchable.

Image display apparatus 103 according to this embodiment thus performs aprocess shown in FIG. 26. Video display 303 displays, as a displayedvideo, at least one of videos that show a scene viewed from differentviewpoints and that include at least one virtual video generated fromreal video (S901).

A viewpoint changer (UI device 304, UI controller 305, and viewpointvideo selector 302) displays viewpoint control UI 401 (the first userinterface) for selecting the viewpoint of the displayed video, andchanges the viewpoint of the displayed video according to an inputprovided via viewpoint control UI 401 (S902).

A speed changer (UI device 304, UI controller 305, and viewpoint videoselector 302) displays speed control UI 402 (the second user interface)for changing the playing speed and the playing direction of thedisplayed video, and changes the playing speed and the playing directionof the displayed video according to an input provided via speed controlUI 402 (S903).

Embodiment 3

In this embodiment, the process of changing the resolution of an imagedisplayed by image display apparatus 103 will be described. Imagedistribution system 100, image distribution apparatus 102, and imagedisplay apparatus 103 are generally configured as in Embodiment 1.

FIG. 27 is a diagram schematically illustrating an example of switchingthe displayed video in image display apparatus 103. Video display 303 inimage display apparatus 103 displays free-viewpoint video. Image displayapparatus 103 also displays images by switching between low-resolutionimage 501 and high-resolution image 502. For example, low-resolutionimage 501 here is one of the videos in the integrated video described inEmbodiment 1, or one of the images in the integrated image described inEmbodiment 1. High-resolution image 502 is an image or a video seen fromthe same viewpoint as low-resolution image 501 and having a higherresolution than low-resolution image 501.

Instead of displaying images by switching between low-resolution image501 and high-resolution image 502, image display apparatus 103 maydisplay images by overlaying high-resolution image 502 on low-resolutionimage 501.

FIG. 28 is a diagram schematically illustrating an example of switchingthe displayed video in image display apparatus 103. First, image displayapparatus 103 displays an integrated video that includes low-resolutionimages 501, as shown in (a) in FIG. 28. In this state, a user of imagedisplay apparatus 103 selects low-resolution image 501A, for example. Asa result, low-resolution image 501B, which is enlarged low-resolutionimage 501A, is displayed as shown in (b) in FIG. 28. That is,low-resolution image 501B is larger than low-resolution image 501A insize. Then, in response to, e.g., a predetermined operation by the user,low-resolution image 501B is switched to high-resolution image 502. Forexample, high-resolution image 502 has the same size as low-resolutionimage 501B. It is to be noted that low-resolution image 501A may betemporarily enlarged and displayed as low-resolution image 501B, orlow-resolution image 501B may be managed separately from low-resolutionimage 501A.

Although the integrated video includes four low-resolution images in theexample illustrated here, the integrated video may include any number oflow-resolution images other than four. Further, not only one butmultiple low-resolution images may be selected and simultaneouslydisplayed in an enlarged size on a partitioned screen.

Instead of being directly selected in the integrated video,low-resolution image 501A may be selected with, e.g., a UI as describedin the other embodiments. That is, the integrated video may notnecessarily be displayed.

In the example illustrated here, high-resolution image 502 is displayedafter low-resolution image 501B is displayed as in (b) in FIG. 28.Alternatively, low-resolution image 501B may not be displayed andhigh-resolution image 502 may be immediately displayed.

Now, the flow of operations in image display apparatus 103 (a displayterminal) and image distribution apparatus 102 (a server) will bedescribed. FIG. 29 is a flowchart illustrating the operations in imagedisplay apparatus 103 and image distribution apparatus 102.

First, image display apparatus 103 displays an integrated video beingreceived (S1001). The integrated video here includes low-resolutionimages 501. Instead of simply displaying the integrated video as it is,image display apparatus 103 may select one or more of low-resolutionimages 501 in the integrated video and display the selected one or moreof low-resolution images 501.

Image display apparatus 103 determines whether to requesthigh-resolution image 502 from image distribution apparatus 102 (S1002).For example, image display apparatus 103 determines to requesthigh-resolution image 502 if a predetermined operation is performed bythe user. This operation is performed with, for example, a buttonprovided by UI device 304 of image display apparatus 103. As anotherexample, image display apparatus 103 determines to requesthigh-resolution image 502 if the same viewpoint is continuously viewedfor a predetermined time period, or if no viewpoint changing operationnor timepoint changing operation is performed while display is beingpaused.

If image display apparatus 103 determines not to request high-resolutionimage 502 (No at S1002), image display apparatus 103 returns to stepS1001. If image display apparatus 103 determines to requesthigh-resolution image 502 from image distribution apparatus 102 (Yes atS1002), image display apparatus 103 transmits a request signal to imagedistribution apparatus 102 requesting to send high-resolution image 502(S1003). This request signal includes viewpoint information indicatingthe viewpoint of requested high-resolution image 502, or timeinformation indicating the timepoint or time segment of requestedhigh-resolution image 502.

Image display apparatus 103 receives high-resolution image 502transmitted by image distribution apparatus 102 (S1004).

Image display apparatus 103 displays high-resolution image 502 receivedat step S1004 on video display 303 (S1005).

According to viewpoint change or play-timepoint change instructed by,e.g., the user's operation, image display apparatus 103 determineswhether to display a portion other than high-resolution image 502received at step S1004 (S1006). If high-resolution image 502 is to becontinuously displayed and therefore the received data exists (No atS1006), image display apparatus 103 continues displaying high-resolutionimage 502 (S1005). If another portion is to be displayed and thereforeno received data exists (Yes at S1006), image display apparatus 103displays low-resolution images 501 (S1001).

The operations in image distribution apparatus 102 will now bedescribed. Image distribution apparatus 102 receives the request signalfor high-resolution image 502 from image display apparatus 103 (S1101).Image distribution apparatus 102 prepares high-resolution image 502based on the request signal (S1102). For example, if high-resolutionimage 502 is a still image, the request signal includes informationindicating the viewpoint of requested high-resolution image 502 andinformation indicating which timepoint in the video the image is locatedat. If high-resolution image 502 is a video, the request signal includesinformation indicating the viewpoint of requested high-resolution image502. Image distribution apparatus 102 may store high-resolution images502 seen from multiple viewpoints in advance and obtain high-resolutionimage 502 seen from the requested viewpoint among stored high-resolutionimages 502. Alternatively, image distribution apparatus 102 maygenerate, as needed, high-resolution image 502 for the viewpoint andtimepoint indicated in the request signal from original images (cameraimages) or from a three-dimensional model.

Image distribution apparatus 102 transmits high-resolution image 502prepared at step S1102 to image display apparatus 103 (S1103).

If there are images or videos seen from associated viewpoints (which areviewpoints associated with the viewpoint indicated in the request signalreceived at step S1101), image distribution apparatus 102 preparesrelevant data, such as by generating high-resolution images seen fromthe associated viewpoints (S1104).

FIG. 30 is a schematic diagram for describing the associated-viewpointimages at step S1104. For example, as shown in FIG. 30, if viewpoint 504is indicated in the request signal received at step S1101, imagedistribution apparatus 102 determines viewpoints positionally close toviewpoint 504 to be associated viewpoints 505 and 506, and prepareshigh-resolution images 502 seen from associated viewpoints 505 and 506.For example, as shown in FIG. 30, associated viewpoints 505 and 506 areviewpoints adjacent to viewpoint 504, for example viewpoints on bothsides of viewpoint 504.

High-resolution mode, in which high-resolution image 502 is displayed,may allow a viewpoint to be selected with finer granularity than normalmode, in which low-resolution images 501 are displayed. In other words,more viewpoints may be selectable in high-resolution mode than in normalmode. For example, N viewpoints concentrically located around the gazepoint may be selectable in normal mode, whereas 2N viewpointsconcentrically located around the gaze point may be selectable inhigh-resolution mode (e.g., while the video is being stopped).

In high-resolution mode, the frame rate of the video may be increased.For example, in normal mode such as when the video is being distributed,the frame rate of the video (low-resolution images 501) beingdistributed may be 30 fps. By contrast, in high-resolution mode such aswhen the video is stopped, played forward frame by frame, or playedbackwards frame by frame, images may be selected from high-resolutionimages 502 of a frame rate of 60 fps.

In high-resolution mode, indicator 503 may be displayed as shown in FIG.30. User operations may be restricted in this state. For example,operations for resuming the video playing and for switching theviewpoint may be inhibited in high-resolution mode.

In high-resolution mode, not the entire image but only specific area 511in the image may be displayed in high resolution. FIG. 31 is a diagramillustrating an exemplary displayed image in this case. Specific area511 here is a key area in the image, for example an area containing aface or an area containing characters, such as characters on a licenseplate of a car.

For example, image distribution apparatus 102 distributes informationindicating specific area 511 along with the image. Image displayapparatus 103 displays the information indicating specific area 511 as adotted-line rectangular shown in FIG. 31. If the dotted-line rectangularindicating specific area 511 is operated (selected), image displayapparatus 103 displays high-resolution image 512 of specific area 511.Alternatively, image display apparatus 103 may always displayhigh-resolution image 512 of specific area 511.

As shown in FIG. 31, image display apparatus 103 may displayhigh-resolution image 512 in a separate window with an indication of therelationship with specific area 511. Instead of displaying in a separatewindow, image display apparatus 103 may display high-resolution image512 to be overlaid on specific area 511.

For prompting the user's recognition of specific area 511, viewpointcontrol UI 401B shown in FIG. 19 displaying the camera arrangement maypresent a recommended viewpoint that facilitates the view of specificarea 511, or a timepoint for each viewpoint that facilitates the view ofspecific area 511 from that viewpoint. For example, the recommendedviewpoint may be a viewpoint from which vehicles can be captured fromthe front or rear direction in an environment such as a crossing, suchthat vehicle numbers are easily recognized. The recommended viewpointmay also be a viewpoint capable of recognizing the face of a focusperson in a concert or game. The timepoint of a particular scene (suchas a shoot scene) of the focus person may also be presented.

Image display apparatus 103 may display the result of analyzingthree-dimensional information. For example, image distribution apparatus102 may recognize vehicles from three-dimensional information and detectthe speed of each vehicle. Image display apparatus 103 may then displaythe speed information obtained.

Thus, image display apparatus 103 according to this embodiment performsthe process shown in FIG. 32. First, image display apparatus 103receives an integrated image in which low-resolution images 501 of ascene seen from different viewpoints are arranged in a frame (S1201).Image display apparatus 103 then displays at least one of low-resolutionimages 501 included in the received integrated image (S1202).

Image display apparatus 103 receives high-resolution image 502 seen fromthe same viewpoint as and having a higher resolution than a selected oneof low-resolution images 501 (S1203). Image display apparatus 103displays received high-resolution image 502 (S1204).

In this manner, an increase in the amount of data to be distributed canbe prevented using the integrated image in which low-resolution images501 are arranged in a frame. Image display apparatus 103 can displayimages by switching between low-resolution image 501 and high-resolutionimage 502, thereby providing a video of high image quality to the user.Image display apparatus 103 can thus improve usability for the userviewing the video while preventing an increase in the amount of data tobe distributed.

For example, at step S1202, image display apparatus 103 displays theintegrated image ((a) in FIG. 28). At step S1204, image displayapparatus 103 displays high-resolution image 502 in a second size largerthan a first size of the selected low-resolution image as displayed inthe integrated image ((c) in FIG. 28).

For example, after displaying the integrated image, image displayapparatus 103 displays the selected low-resolution image in a third sizelarger than the first size ((b) in FIG. 28). After displaying theselected low-resolution image in the third size, image display apparatus103 displays high-resolution image 502 in the second size ((c) in FIG.28).

For example, if the user's predetermined operation is received, imagedisplay apparatus 103 transitions from normal mode in whichlow-resolution images 501 are received and displayed to high-resolutionmode in which high-resolution image 502 is received and displayed.

For example, the selected low-resolution image is a video, and thepredetermined operation is an operation for pausing the selectedlow-resolution image. This allows providing an image of high imagequality to the user while preventing an increase in the amount of datato be distributed.

For example, the selected low-resolution image is a video, and thepredetermined operation is a frame-by-frame advance operation or aslow-motion replay operation for the selected low-resolution image. Inhigh-resolution mode, image display apparatus 103 receives ahigh-resolution image having a higher frame rate than the selectedlow-resolution image, and displays the received high-resolution imageframe by frame or plays the received high-resolution image at a slowspeed. This allows providing an image or a video of high image qualityto the user in high-resolution mode.

For example, in high-resolution mode, more viewpoints are selectablethan in normal mode. This allows increasing the number of viewpointsselectable by the user in high-resolution mode.

For example, as shown in FIG. 31, high-resolution image 512 is ahigh-resolution image of a partial area (specific area 511) in theselected low-resolution image. This allows only a partial area to bedisplayed in high resolution, thereby preventing an increase in theamount of data. For example, the partial area is an area containing aface or characters.

Image distribution apparatus 102 according to this embodiment performsthe process shown in FIG. 33. First, image distribution apparatus 102generates an integrated image in which low-resolution images 501 of ascene seen from different viewpoints are arranged in a frame (S1301).Image distribution apparatus 102 then distributes the generatedintegrated image to image display apparatus 103 (S1302).

In response to a request from image display apparatus 103, imagedistribution apparatus 102 generates high-resolution image 502 seen fromthe same viewpoint as and having a higher resolution than a selected oneof low-resolution images 501 included in the integrated image (S1303).Image distribution apparatus 102 distributes generated high-resolutionimage 502 to image display apparatus 103 (S1304).

In this manner, an increase in the amount of data to be distributed canbe prevented using the integrated image in which low-resolution images501 are arranged in a frame. Image display apparatus 103 can displayimages by switching between low-resolution image 501 and high-resolutionimage 502, thereby providing a video of high image quality to the user.Image distribution apparatus 102 can thus improve usability for the userviewing the video while preventing an increase in the amount of data tobe distributed.

For example, if image display apparatus 103 receives the user'spredetermined operation, image distribution apparatus 102 transitionsfrom normal mode in which low-resolution images 501 are distributed tohigh-resolution mode in which high-resolution image 502 is distributed.

For example, the selected low-resolution image is a video, and thepredetermined operation is an operation for pausing the selectedlow-resolution image. This allows providing an image of high imagequality to the user while preventing an increase in the amount of datato be distributed.

For example, the selected low-resolution image is a video, and thepredetermined operation is a frame-by-frame advance operation or aslow-motion replay operation for the selected low-resolution image. Inhigh-resolution mode, image distribution apparatus 102 distributeshigh-resolution image 502 having a higher frame rate than the selectedlow-resolution image to image display apparatus 103. This allowsproviding an image or a video of high image quality to the user inhigh-resolution mode.

For example, in high-resolution mode, more viewpoints are selectable bythe user than in normal mode. This allows increasing the number ofviewpoints selectable by the user in high-resolution mode.

For example, as shown in FIG. 31, high-resolution image 512 is ahigh-resolution image of a partial area (specific area 511) in theselected low-resolution image. This allows only a partial area to bedisplayed in high resolution, thereby preventing an increase in theamount of data. For example, the partial area is an area containing aface or characters.

For example, as shown in FIG. 30, when distributing high-resolutionimage 502 corresponding to the selected low-resolution image to imagedisplay apparatus 103, image distribution apparatus 102 generates animage that is seen from a viewpoint close to the viewpoint of theselected low-resolution image among the viewpoints of low-resolutionimages 501 and that has a higher resolution than the low-resolutionimages. This allows reducing the time before video is displayed afterthe viewpoint is switched.

Although an image distribution system, an image distribution apparatus,and an image display apparatus according to exemplary embodiments of thepresent disclosure have been described above, the present disclosure isnot limited to such embodiments.

Note that each of the processing units included in the imagedistribution system according to the embodiments is implementedtypically as a large-scale integration (LSI), which is an integratedcircuit (IC). They may take the form of individual chips, or one or moreor all of them may be encapsulated into a single chip.

Furthermore, the integrated circuit implementation is not limited to anLSI, and thus may be implemented as a dedicated circuit or ageneral-purpose processor. Alternatively, a field programmable gatearray (FPGA) that allows for programming after the manufacture of anLSI, or a reconfigurable processor that allows for reconfiguration ofthe connection and the setting of circuit cells inside an LSI may beemployed.

Moreover, in the above embodiments, the structural components may beimplemented as dedicated hardware or may be realized by executing asoftware program suited to such structural components. Alternatively,the structural components may be implemented by a program executor suchas a CPU or a processor reading out and executing the software programrecorded in a recording medium such as a hard disk or a semiconductormemory.

Furthermore, the present disclosure may be embodied as various methodsperformed by the image distribution system, the image distributionapparatus, or the image display apparatus.

Furthermore, the divisions of the blocks shown in the block diagrams aremere examples, and thus a plurality of blocks may be implemented as asingle block, or a single block may be divided into a plurality ofblocks, or one or more blocks may be combined with another block. Also,one or more blocks may be implemented in single hardware or by software.

Furthermore, the processing order of executing the steps shown in theflowcharts is a mere illustration for specifically describing thepresent disclosure, and thus may be an order other than the shown order.Also, one or more of the steps may be executed simultaneously (inparallel) with another step.

Although the image distribution system according to one or more aspectshas been described on the basis of the exemplary embodiments, thepresent disclosure is not limited to such embodiments. The one or moreaspects may thus include forms obtained by making various modificationsto the above embodiments that can be conceived by those skilled in theart, as well as forms obtained by combining structural components indifferent embodiments, without materially departing from the spirit ofthe present disclosure.

INDUSTRIAL APPLICABILITY

The present disclosure is applicable to image distribution systems andimage distribution methods, for example, free-viewpoint video generationsystems and next-generation monitoring systems.

What is claimed is:
 1. An image display method comprising; receiving anintegrated image in which low-resolution images of a scene fromdifferent viewpoints are arranged; displaying the integrated image suchthat a low-resolution image among the low-resolution images is displayedin a first size; receiving a high-resolution image having a sameviewpoint as the low-resolution image, the high-resolution image havinga higher resolution than the low-resolution image; displaying thehigh-resolution image in a second size larger than the first size; anddisplaying the low-resolution image in a third size larger than thefirst size after the integrated image is displayed and before thehigh-resolution image is displayed in the second size.
 2. The imagedisplay method according to claim 1, further comprising: selecting thelow-resolution image from among the low-resolution images according toan operation by a user.
 3. The image display method according to claim1, wherein the second size is same as the third size.
 4. The imagedisplay method according to claim 1, comprising; transitioning fromnormal mode in which the low-resolution images are received anddisplayed to high-resolution mode in which the high-resolution image isreceived and displayed, when a predetermined operation by a user isreceived.
 5. The image display method according to claim 4, wherein thelow-resolution image is included in a first video, and the predeterminedoperation is an operation for pausing the first video.
 6. The imagedisplay method according to claim 4, wherein the low-resolution image isincluded in a first video, the predetermined operation is aframe-by-frame advance operation or a slow-motion replay operation forthe first video, and in the high-resolution mode, a second videoincluding the high-resolution image and having a frame rate higher thana frame rate of the first video is received, and displayed frame byframe or played at a slow speed.
 7. The image display method accordingto claim 4, wherein more viewpoints are selectable in thehigh-resolution mode than in the normal mode.
 8. An image display methodcomprising: receiving an integrated image in which low-resolution imagesof a scene from different viewpoints are arranged; displaying at leastone of the low-resolution images; receiving a high-resolution imagewhich shows a partial area of a low-resolution image among thelow-resolution images and which has a higher resolution than thelow-resolution image; and displaying the high-resolution image.
 9. Theimage display method according to claim 8, wherein the partial areashows a face or characters.
 10. An image distribution method comprising:generating an integrated image in which low-resolution images of a scenefrom different viewpoints are arranged; distributing the integratedimage to an image display apparatus; generating, in response to arequest from the image display apparatus, a high-resolution image havinga same viewpoint as a low-resolution image among the low-resolutionimages, the high-resolution image having a higher resolution than thelow-resolution image; and distributing, to the image display apparatus,the high-resolution image generated.
 11. The image distribution methodaccording to claim 10, comprising: transitioning from normal mode inwhich the low-resolution images are distributed to high-resolution modein which the high-resolution image is distributed, when the imagedisplay apparatus receives a predetermined operation by a user.
 12. Theimage distribution method according to claim 11, wherein thelow-resolution image is included in a first video, and the predeterminedoperation is an operation for pausing the first video.
 13. The imagedistribution method according to claim 11, wherein the low-resolutionimage is included in first a video, the predetermined operation is aframe-by-frame advance operation or a slow-motion replay operation forthe first video, and in the high-resolution mode, a second videoincluding the high-resolution image and having a frame rate higher thana frame rate of the first video is distributed to the image displayapparatus.
 14. The image distribution method according to claim 11,wherein more viewpoints are selectable by the user in thehigh-resolution mode than in the normal mode.
 15. The image distributionmethod according to claim 10, wherein the high-resolution image shows apartial area in the low-resolution image.
 16. The image distributionmethod according to claim 15, wherein the partial area shows a face orcharacters.
 17. The image distribution method according to claim 10,comprising: when distributing the high-resolution image corresponding tothe low-resolution image to the image display apparatus, generating animage which is from a viewpoint close to the viewpoint of thelow-resolution image among the viewpoints of the low-resolution imagesand has a higher resolution than the low-resolution images.
 18. Theimage distribution method according to claim 10, further comprising:selecting the low-resolution image from among the low-resolution imagesaccording to an operation by a user.
 19. An image display apparatuscomprising: circuitry; and memory, wherein the circuitry, using thememory, performs the image display method according to claim
 1. 20. Animage distribution apparatus comprising: circuitry; and memory, whereinthe circuitry, using the memory, performs the image distribution methodaccording to claim 10.